Towards a Mini-App for Smoothed Particle Hydrodynamics at Exascale

The smoothed particle hydrodynamics (SPH) technique is a purely Lagrangian method, used in numerical simulations of fluids in astrophysics and computational fluid dynamics, among many other fields. SPH simulations with detailed physics represent computationally-demanding calculations. The parallelization of SPH codes is not trivial due to the absence of a structured grid. Additionally, the performance of the SPH codes can be, in general, adversely impacted by several factors, such as multiple time-stepping, long-range interactions, and/or boundary conditions. This work presents insights into the current performance and functionalities of three SPH codes: SPHYNX, ChaNGa, and SPH-flow. These codes are the starting point of an interdisciplinary co-design project, SPH-EXA, for the development of an Exascale-ready SPH mini-app. To gain such insights, a rotating square patch test was implemented as a common test simulation for the three SPH codes and analyzed on two modern HPC systems. Furthermore, to stress the differences with the codes stemming from the astrophysics community (SPHYNX and ChaNGa), an additional test case, the Evrard collapse, has also been carried out. This work extrapolates the common basic SPH features in the three codes for the purpose of consolidating them into a pure-SPH, Exascale-ready, optimized, mini-app. Moreover, the outcome of this serves as direct feedback to the parent codes, to improve their performance and overall scalability.Comment: 18 pages, 4 figures, 5 tables, 2018 IEEE International Conference on Cluster Computing proceedings for WRAp1

Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface and the HDF5 DSM Virtual File Driver

Honourable Mention AwardInternational audienceWe present a framework for interfacing an arbitrary HPC simulation code with an interactive ParaView session using the HDF5 parallel IO library as the API. The implementation allows a flexible combination of parallel simulation, concurrent parallel analysis and GUI client, all of which may be on the same or separate machines. Data transfer between the simulation and the ParaView server takes place using a virtual file driver for HDF5 that bypasses the disk entirely and instead communicates directly between the coupled applications in parallel. The simulation and ParaView tasks run as separate MPI jobs and may therefore use different core counts and/or hardware configurations/platforms, making it possible to carefully tailor the amount of resources dedicated to each part of the workload. The coupled applications write and read datasets to the shared virtual HDF5 file layer, which allows the user to read data representing any aspect of the simulation and modify it using ParaView pipelines, then write it back, to be reread by the simulation (or vice versa). This allows not only simple parameter changes, but complete remeshing of grids, or operations involving regeneration of field values over the entire domain, to be carried out. To avoid the problem of manually customizing the GUI for each application that is to be steered, we make use of XML templates that describe outputs from the simulation, inputs back to it, and what user interactions are permitted on the controlled elements. This XML is used to generate GUI and 3D controls for manipulation of the simulation without requiring explicit knowledge of the underlying model

SPH-EXA: Enhancing the Scalability of SPH codes Via an Exascale-Ready SPH Mini-App

Numerical simulations of fluids in astrophysics and computational fluid dynamics (CFD) are among the most computationally-demanding calculations, in terms of sustained floating-point operations per second, or FLOP/s. It is expected that these numerical simulations will significantly benefit from the future Exascale computing infrastructures, that will perform 10^18 FLOP/s. The performance of the SPH codes is, in general, adversely impacted by several factors, such as multiple time-stepping, long-range interactions, and/or boundary conditions. In this work an extensive study of three SPH implementations SPHYNX, ChaNGa, and XXX is performed, to gain insights and to expose any limitations and characteristics of the codes. These codes are the starting point of an interdisciplinary co-design project, SPH-EXA, for the development of an Exascale-ready SPH mini-app. We implemented a rotating square patch as a joint test simulation for the three SPH codes and analyzed their performance on a modern HPC system, Piz Daint. The performance profiling and scalability analysis conducted on the three parent codes allowed to expose their performance issues, such as load imbalance, both in MPI and OpenMP. Two-level load balancing has been successfully applied to SPHYNX to overcome its load imbalance. The performance analysis shapes and drives the design of the SPH-EXA mini-app towards the use of efficient parallelization methods, fault-tolerance mechanisms, and load balancing approaches.Comment: arXiv admin note: substantial text overlap with arXiv:1809.0801

Parallel Computational Steering for HPC Applications Using HDF5 Files in Distributed Shared Memory

International audienceInterfacing a GUI driven visualization/analysis package to an HPC application enables a supercomputer to be used as an interactive instrument. We achieve this by replacing the IO layer in the HDF5 library with a custom driver which transfers data in parallel between simulation and analysis. Our implementation using ParaView as the interface, allows a flexible combination of parallel simulation, concurrent parallel analysis, and GUI client, either on the same or separate machines. Each MPI job may use different core counts or hardware configurations, allowing fine tuning of the amount of resources dedicated to each part of the workload. By making use of a distributed shared memory file, one may read data from the simulation, modify it using ParaView pipelines, write it back, to be reused by the simulation (or vice versa). This allows not only simple parameter changes, but complete remeshing of grids, or operations involving regeneration of field values over the entire domain. To avoid the problem of manually customizing the GUI for each application that is to be steered, we make use of XML templates that describe outputs from the simulation (and inputs back to it) to automatically generate GUI controls for manipulation of the simulation

Parallélisation d'un code SPH 3D pour des simulations massives sur mémoire distribuée

La méthode Smoothed Particle Hydrodynamics (SPH) est une méthode particulaire qui a connu une forte émergence au cours des deux dernières décennies. Bien qu’initialement développée pour des applications astrophysiques, cette méthode numérique est aujourd'hui largement appliquée à la mécanique des fluides, à la mécanique des structures et à des applications variées dans différentes branches de la physique. Le code SPH-flow est développé conjointement par le LHEEA de l'Ecole Centrale de Nantes et l’entreprise HydrOcean. Cet outil est principalement dédié à la modélisation d’écoulements complexes à surface libre, dans un contexte de dynamique rapide, et en présence de solides présentant des géométries complexes en interaction avec le fluide. Dans ce domaine, le principal avantage de cette méthode repose sur sa capacité à simuler les déconnexions/reconnexions de surface libre (déferlements, jets de surface libre...) sans nécessiter sa capture. Comme c’est le cas pour la plupart des autres méthodes particulaires, cette méthode est exigeante en termes de ressources de calcul, et sa parallélisation est inévitable dans le cadre d’applications 3D massives, en vue de conserver des temps de restitution raisonnables. L'ordre de grandeur des résolutions adoptées dans nos simulations est de plusieurs centaines de millions de particules, impliquant l’utilisation de plusieurs milliers de processeurs en réseau, et donc une parallélisation performante. Cet article présente la stratégie de parallélisation retenue dans notre code SPH, basée sur une décomposition de domaine et passant par l’utilisation du standard MPI (mémoire distribuée). Les différentes procédures dédiées à l'équilibrage de charge sont présentées, ainsi que les solutions retenues pour limiter les latences de communication par l’emploi de communication non bloquantes. Les performances parallèles sont ensuite présentées en termes d’accélération et d’efficacité, sur des cas allant jusqu’à 3 milliards de particules et utilisant 32768 processeurs en réseau

SPH-EXA: Enhancing the Scalability of SPH codes Via an Exascale-Ready SPH Mini-App

Numerical simulations of fluids in astrophysics and computational fluid dynamics (CFD) are among the most computationally-demanding calculations, in terms of sustained floating-point operations per second, or FLOP/s. It is expected that these numerical simulations will significantly benefit from the future Exascale computing infrastructures, that will perform 10^18 FLOP/s. The performance of the SPH codes is, in general, adversely impacted by several factors, such as multiple time-stepping, long-range interactions, and/or boundary conditions. In this work an extensive study of three SPH implementations SPHYNX, ChaNGa, and XXX is performed, to gain insights and to expose any limitations and characteristics of the codes. These codes are the starting point of an interdisciplinary co-design project, SPH-EXA, for the development of an Exascale-ready SPH mini-app. We implemented a rotating square patch as a joint test simulation for the three SPH codes and analyzed their performance on a modern HPC system, Piz Daint. The performance profiling and scalability analysis conducted on the three parent codes allowed to expose their performance issues, such as load imbalance, both in MPI and OpenMP. Two-level load balancing has been successfully applied to SPHYNX to overcome its load imbalance. The performance analysis shapes and drives the design of the SPH-EXA mini-app towards the use of efficient parallelization methods, fault-tolerance mechanisms, and load balancing approaches

SPH-EXA: Optimizing Smoothed Particle Hydrodynamics for Exascale Computing

The SPH-EXA project, funded by the Swiss Platform for Advanced Scientific Computing, brings together expertise in High Performance Computing, Astrophysics, and Computational Fluid Dynamics. The Smoothed Particle Hydrodynamics (SPH) technique is a purely Lagrangian method, used in numerical simulations of fluids in astrophysics and computational fluid dynamics, among many other fields. SPH simulations are among the most computationally-demanding calculations, in terms of sustained floating-point operations per second, or FLOP / s. It is expected that these numerical simulations will significantly benefit from the future Exascale computing infrastructures, that will perform 1018 FLOP / s. The performance of the SPH codes is, in general, adversely impacted by several factors, such as multiple time-stepping, long-range interactions, and / or boundary conditions. SPHYNX, ChaNGa, and SPH-flow, three SPH implementations, are the starting point of an interdisciplinary co-design project, SPH-EXA, for the development of an Exascale-ready SPH mini-app. The goal is to understand the scalability limitations of state-of-the-art SPH implementations and use such knowledge to create an Exascale-ready SPH mini-app. An extensive study of SPHYNX, ChaNGa, and SPH-flow has been performed, to gain insights and to expose the characteristics and any limitations of the codes. Such exposed limitations have been solved in the parent codes and shape the design and implementation of the SPH-EXA mini-app towards the use of efficient parallelization methods, fault-tolerance mechanisms, and load balancing approaches, to sustain the scalable execution of massively parallel SPH codes. The SPH-EXA mini-app development targets reproducibility and portability by using continuous integration systems, and containerization techniques

Large-Scale System Monitoring Experiences and Recommendations Workshop paper: HPCMASPA 2018

Monitoring of High Performance Computing (HPC) platforms is critical to successful operations, can provide insights into performance-impacting conditions, and can inform methodologies for improving science throughput. However, monitoring systems are not generally considered core capabilities in system requirements specifications nor in vendor development strategies. In this paper we present work performed at a number of large-scale HPC sites towards developing monitoring capabilities that fill current gaps in ease of problem identification and root cause discovery. We also present our collective views, based on the experiences presented, on needs and requirements for enabling development by vendors or users of effective sharable end-to-end monitoring capabilities